What is the nature of decision noise in random exploration?

The explore-exploit tradeoff is a fundamental behavioral dilemma faced by all adaptive organisms. Should we explore new options in the hopes of finding a better meal, a better house or a better mate, or should we exploit the options we currently believe to be best? Striking the right balance between exploration and exploitation is hard computational problem and there is significant interest in how humans and other animals make explore-exploit decisions in practice. One particularly effective strategy for solving the explore-exploit dilemma is choice randomization[1]. In this strategy, the decision process is corrupted by noise meaning that high value “exploit” options are not always chosen and exploratory choices are sometimes made by chance. In theory, such “random exploration”, can be surprisingly effective in explore-exploit problems and, if implemented correctly, can come close to optimal performance. Recent work suggests that humans actually use random exploration to solve simple explore-exploit problems[2]. Despite this progress a number of questions remain about the nature of random exploration as there are a number of ways in which seemingly stochastic choices could be generated. In one strategy, that we call the “external noise strategy”, participants could rely on stochasticity in the world and allow irrelevant features of the stimulus to drive choice. In another strategy called “internal noise strategy”, people could rely on stochastic processes within their own brains. In this work, we modified our recently published “Horizon Task” in such a way as to distinguish these two strategies. Using both a model-free and model-based analysis of human behavior we show that both types of noise are present in explore-exploit decisions, but that random exploration is dominated by internal noise. This suggests that random exploration depends on adaptive noise processes in the brain which are subject to (perhaps unconscious) cognitive control.